Extracted this static function for code reuse between the old synchronous code path and the new scene index which gets scene index prims without any conversions.

The new scene index implementation, which replaces materialXSyncDispatcher in the original PR.

I propose to rename _networkProcessor to _hdStMaterialNetwork. IMHO "Processor" sounds confusing here because the object _hdStMaterialNetwork is simply initialized with those Process methods from hdNetworkMap, it doesn't process some external objects.

Now need to call it from outside this source file.

ProcessMaterialNetwork is now implemented via ProcessFilterTask for code reuse.

The state necessary for the codegen process is now encapsulated in the HdSt_MaterialFilterTask class, because it needs to be created earlier than the material sprim and to persist until the parallel codegen task is finished.

HdStMaterialNetwork::ProcessMaterialNetwork is called in the old, synchronous code path, but we create a HdSt_MaterialFilterTask here in order to share code between the two code paths.

This and a few other functions are now methods of HdSt_MaterialFilterTask.

This and a few other related pieces of data are now encapsulated as members of HdSt_MaterialFilterTask.

Unified the terminology around "anonymized". The input parameters are replaced by HdSt_MaterialFilterTask's members.

This is a different kind of "task" - the necessary data passed to the lambda executing in parallel in TBB and wrapping the MaterialX codegen.

Creating the generator task on the stack in the case when parallel codegen is off.

Creating the generator task on the heap for parallel codegen.

One important change from the original implementation is that the generator task now gets a shared pointer to the filter task and holds on to it. The generation process reads some material network data which is part of the filter task, so it's important to ensure that it's not destroyed before codegen is complete. In the original implementation, the sprim held ownership of the filter task for the duration of codegen.

Replaces pxr/imaging/hdSt/materialXSyncDispatcher.cpp in the original implementation

First check that the shader hasn't been registered before the early parallel init. This is a new method which, unlike RegisterMaterialXShader, doesn't add a new instance to the registry if an existing one isn't found.

Reusing RegisterMaterialXShader in combination with IsFirstInstance would be problematic because the intended usage pattern, in the case when there's no existing value, seems to be:

1. Insert a nullptr value and return a registry instance pointing to that value and holding a lock to the whole MaterialX registry.
2. The client code is supposed to populate the instance with a non-nullptr value while holding the lock. The instance's IsFirstInstance method returns true if the value is nullptr.

So the above pattern can't be used for parallel tasks, when the uniqueness check has to happen before launching the task, the non-nullptr value is known only when the task completes and the lock has to be released ASAP.

Makes sense.

Another uniqueness check, this time between the different parallel tasks, using a dedicated data structure.

Moved this setting definition here for better encapsulation.

Injecting our scene index as the very last in the chain, and only if the optimization is enabled.

This is currently only used in HdStMaterial::Sync to wait for the parallel codegen, but could be used in the future to reuse some per-material state.

The new method just wraps existing functionality.

Wait for the codegen tasks to complete. Previously, we were waiting by calling a new render delegate API before syncing the materials: https://github.com/PixarAnimationStudios/OpenUSD/pull/3567/files#r1995705092

Now, we wait for the generator task group in each material's sync. In practice, only the first material sprim would ever wait for any measurable duration, and the overhead of waiting for all subsequent sprims is negligible (around 1 microsecond).

This is also where we get the cached per-material state from the scene index. The filter task is removed from the scene index its ownership is transferred to the local variable.

Reusing the filter task allows us to avoid getting the material resource again below.

Is it possible instead to have the material prim datasource call wait on the scene index, so that Storm doesn't need to deal with the new scene index at all? I'll try to sketch out a proposal in the scene index code.

Once the material network has processed the filter task, we don't need the task any more, so we destroy it by letting it go out of scope.

Deleting the filter task on dirtying.

Caching the filter task in the scene index.

Is it possible instead to have the material prim datasource call wait on the scene index, so that Storm doesn't need to deal with the new scene index at all? I'll try to sketch out a proposal in the scene index code.

Here & material.*, instead of bringing the "filterTask" API into a bunch of files in Storm, you could modularize things better by either:
(1) having the scene index produce a "glsl" material network that can be read by the non-materialx code; or
(2) having the scene index produce a digested material network as code snippets for each stage & params, and pass them through as datasources. If material.cpp finds that, it uses it, and otherwise it falls back to network processing.

This splits the code up a little better, which is nice since the MaterialX code is an external dependency with an evolving API and hidden behind a build flag, so having a big API interface between the two makes me a bit worried.

Don't think this needs an extern, but if you want it to be publicly visible (which maybe we don't care?) you could throw an HDST_API on it...

This feels like an implementation detail of the MaterialX code, and I don't think this belongs in hdSt/materialNetwork.h. Also curious why we need ARCH_EXPORT_TYPE here.

I know you're improving on the status quo here (which is to pass in resource registry and pull hgi out of that), but it would be cleaner to just pass in the relevant info, namely bindless support and shading language.

Makes sense.

I was envisioning that you call WaitAndExtractFilterTask here, either when you call GetPrim() on a materialx prim, or potentially when you call primDataSource->Get("material"). That way, you don't need all of the complicated logic in renderDelegate.h to pass the scene index pointer through.

As mentioned in the scene index, this bit breaks a bunch of encapsulation. We need GetMaterialXSyncSceneIndex so we can call WaitAndExtractFilterTask in material.cpp, but if we do that in the scene index GetPrim() or a datasource instead we don't need this call anymore. Meanwhile, rather than grafting the new scene index onto the terminal scene index like this, the more flexible/preferred way is to register a scene index plugin that inserts it somewhere relative to other scene index filters; it's a small bit of boilerplate you can grab from e.g. the dependencySceneIndexPlugin.* or something.

Then the only issue is your scene index's use of the render delegate, but it shouldn't really be getting a pointer to the render delegate anyway. It does need a pointer to a registry of MaterialX codegen results, but we could either find a lower key way to pass in the resource registry, or just have a local registry on the scene index instead.